Video encoding/decoding method and apparatus

ABSTRACT

A video encoding apparatus comprises a frame memory/predictive image generator having a first predictive mode for generating a predictive image signal by selecting a combination from among a plurality combinations of a reference image number and a plurality of predictive parameters, and a second predictive mode for generating a predictive image signal according to a predictive parameter computed based on reference image numbers of reference images and an image-to-image distance, and a variable-length encoder to select one of the first and second prediction modes by the number of reference images, and encode orthogonal transformation coefficient information concerning a predictive error signal of a predictive image signal with respect to input video signal, mode information indicating an encoding mode, motion vector information and combination of selected reference image number with predictive parameter index information indicating combination of selected reference image information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of and claims priority under 35 USC §120 from application Ser. No. 11/685,565, filed Mar. 13, 2007, theentire contents of which are incorporated herein by reference.Application Ser. No. 11/685,565 is a divisional of application Ser. No.10/896,890, filed Jul. 23, 2004, and is a Continuation Application ofPCT Application No. PCT/JP03/14658, filed Nov. 18, 2003, which was notpublished under PCT Article 21(2) in English.

This application is based upon and claims the benefit of priority fromprior Japanese Patent Application No. 2002-339931, filed Nov. 22, 2002,the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to a method of encoding/decoding effectively afading image and a dissolve image and an apparatus therefor.

2. Description of the Related Art

In a video encoding standard scheme such as ITU-TH.261, H.263,ISO/IECMPEG-2, and MPEG-4, a motion compensated predictive interframeencoding is used as one of encoding modes. As a predictive model in themotion compensated predictive interframe encoding is adopted a modelindicating the most high predictive efficiency when luminosity does notvary in a time axis. In the case of the fading image that luminosity ofthe image varies, for example, in the case of fading-in from a blackimage to a normal image, a method of performing a prediction adequatelyaccording to a change of luminosity of the image is not known.Consequently, there is a problem to need the large number of encodedbits for the purpose of maintaining an image quality in a fading image.

fading-in from a black image to a normal image, a method of performing aprediction adequately according to a change of luminosity of the imageis not known. Consequently, there is a problem to need the large numberof encoded bits for the purpose of maintaining an image quality in afading image.

For example, U.S. Pat. No. 3,166,716 discloses a technique of copingwith the problem by detecting a fading image area and changing anallocation of the number of encoded bits thereto. Concretely, in thecase of the fade-out image, the large number of encoded bits areallocated to an beginning part of the fade-out that brightness varies.The last part of the fade-out usually reduces allocation of the numberof encoded bits since it normally becomes a monochrome image resultingin making the encoding easy. With such a way, a total image quality isimproved without increasing the total number of encoded bits.

On the other hand, U.S. Pat. No. 2,938,412 discloses an encoding systemthat deals with the above problem in a fading image by compensating fora reference image according to two parameters of an amount of luminancechange and an amount of contrast change.

Thomas Wiegand and Berand Girod, “Multi-frame motion-compensatedprediction for video transmission”, Kluwer Academic Publishers 2001,provides an encoding system based on a plurality of frame buffers. Thissystem intends to improve a predictive efficiency by generating apredictive image selectively from a plurality of reference frames savedin the frame buffers.

However, the system of U.S. Pat. No. 3,166,716 improves an image qualitywithout increasing the total number of encoded bits in encoding a fadingimage by detecting a fading image area, and changing an allocation ofthe number of encoded bits. For this reason, there is the advantage thatthe encoding can be realized within the framewark of an existingencoding system. However, since the prediction efficiency is notessentially improved, a notable improvement of the encoding efficiencycannot be expected.

On the other hand, the system of U.S. Pat. No. 2,938,412 has a meritthat the predictive efficiency on a fading image improves. However, thepredictive efficiency to be enough for so-called a dissolve image(referred to as a cross fade image) which an image gradually varies froman image to another image cannot be obtained.

The system of Thomas Wiegand and Berand Girod cannot sufficiently dealwith a fading image and a dissolve image, and cannot improve apredictive efficiency even if a plurality of reference frames areprepared.

According to the prior art as described above, the large number ofencoded bits are required for the fading image and dissolve image to beencoded with high image quality. There is a problem that improvement ofthe encoding efficiency cannot be expected.

BRIEF SUMMARY OF THE INVENTION

An object of the present invention is to provide a video-encodingmethod, a video-decoding method, and an apparatus therefor that make itpossible to encode in high efficiency a video image whose luminancevaries with time such as a fading image and a dissolve image anddecrease a computation amount.

In a first aspect of the present invention, in the case of subjecting aninput video signal to a motion compensated predictive encoding on avideo encoding side by using at least one reference image signal and amotion vector between the input video signal and the reference imagesignal, if the number of reference images used for a motion compensatedpredictive encoding is single, there is used a first predictive imagesignal generation method of generating a predictive image signalaccording to a reference image number and a predictive parameter of acombination selected for each to-be-encoded region of the input videosignal, from a plurality of combinations of at least one reference imagenumber with predictive parameters.

On the other hand, when the number of reference images used for a motioncompensated predictive encoding is plural, there is used a secondpredictive signal generation method of generating a predictive imagesignal according to a predictive parameter computed every to-be-encodedregion based on the reference image numbers of the plurality ofreference images and an image-to-image distance of the plurality ofreference images.

A predictive error signal expressing an error of a predictive imagesignal generated in this way with respect to an input video signal isgenerated, and index information indicating either of a set of thepredictive error signal and motion vector information and a set of aselected combination and the reference image numbers of a plurality ofreference images is encoded.

In another aspect of the present invention, when a predictive type of ato-be-encoded region of an input video signal is a first predictive typeto use a reference image for a motion compensated predictive encoding, afirst prediction signal generation method is used. When a predictivetype of a to-be-encoded region is a bidirectional predictive type andthe number of reference images used for a motion compensated predictiveencoding is plural, a second predictive signal generation method isused.

On the other hand, on a video decoding side, coded data including apredictive error signal representing an error of a predictive imagesignal with respect to a video signal, motion vector information, andindex information indicating either of a combination of the referenceimage number and a predictive parameter and the reference image numbersof a plurality of reference images are decoded. When the decoded indexinformation represents the combination, a predictive image signal isgenerated according to the reference image number and predictiveparameter of the combination. When the decoded index informationrepresents the reference image number of a plurality of referenceimages, a predictive image signal is generated according to thereference image number and a predictive parameter computed based on animage-to-image distance of the plurality of reference images. A playbackvideo signal is generated using the predictive error signal andpredictive image signal generated in this way.

According to the present invention as thus described, a first predictiveimage generation method of generating a predictive image signalaccording to a combination of the reference image number and apredictive parameter, and a second predictive image generation method ofgenerating a predictive image signal using a predictive parametercalculated based on a frame-to-frame distance of selected plurality ofreference images are prepared, and either thereof is selected accordingto the number of reference images and a predictive type that are usedfor a motion compensated predictive encoding.

As a result, an appropriate predictive image signal can be generated bya predictive system with a higher predictive efficiency for an inputvideo signal from which a predictive image signal cannot beappropriately generated by a predictive system of a normal videoencoding, for example, a fading image and a dissolve image, too.

In addition, since it is possible to set the number of times ofmultiplication per a pixel to one time, a hardware scale and anoperation cost can be reduced in both of the encoding side and thedecoding side.

Further, information concerning a reference image number and apredictive parameter is not sent from the encoding side to the decodingside, but index information indicating combination of a reference imagenumber with a predictive parameter, or index information indicatingcombination of predictive parameters is sent in the case of sending thereference image number separately. As a result, the encoding efficiencycan be improved.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

FIG. 1 is a block diagram showing configuration of a video encodingapparatus relating to a first embodiment of the present invention.

FIG. 2 is a block diagram showing detailed configuration of a framememory and a predictive image generator in the embodiment.

FIG. 3 is a diagram showing an example of a combination table accordingto a reference image number and a predictive parameter to use in theembodiment.

FIG. 4 is a diagram showing a first position relation between tworeference images and a to-be-encoded image in the embodiment.

FIG. 5 is a diagram showing a second position relational between tworeference images and a to-be-encoded image in the embodiment.

FIG. 6 is a diagram showing a third position between two referenceimages and a to-be-encoded image in the embodiment.

FIG. 7 is a diagram showing a fourth position relation between tworeference images and a to-be-encoded image in the embodiment.

FIG. 8 is a flowchart showing an example of a procedure of a selectionof a predictive scheme every macroblock (combination of a referenceimage number with a predictive parameter) and an encoding modedetermination.

FIG. 9 is a block diagram showing configuration of a video decodingapparatus relative to the embodiment.

FIG. 10 is a block diagram illustrating a detailed configuration of aframe memory/predictive image generator in FIG. 9.

FIG. 11 is a diagram showing an example of syntax every block in thecase of encoding index information.

FIG. 12 is a diagram showing an example of a concrete encoded bit streamin the case of generating a predictive image using a single referenceimage.

FIG. 13 is a diagram showing an example of a concrete encoded bit streamin the case of generating a predictive image using two reference images.

FIG. 14 is a flowchart showing a procedure to change a predictive schemeaccording to a kind of to-be-encoded region relative to an embodiment ofthe present invention.

DETAILED DESCRIPTION OF THE INVENTION

An embodiment of the present invention will now be described withreference to drawings.

First Embodiment With Reference to the Encoding Side

FIG. 1 shows configuration of a video encoding apparatus related to thefirst embodiment of the present invention. To the video encodingapparatus is input a video signal 100 in units of a frame, for example,in this example. This video signal 100 is input to a subtracter 101 togenerate a predictive error signal by obtaining a difference withrespect to a predictive image signal 212. Either of the predictive errorsignal and input video signal 100 is selected with a mode selectionswitch 102, and subjected to an orthogonal transformation, for example,discrete cosine transform (DCT) with an orthogonal transformer 103. Theorthogonal transformer 103 generates orthogonal transformed coefficientinformation, for example, DCT coefficient information. The orthogonaltransformed coefficient information is quantized with a quantizer 104,and then quantized orthogonal transformed coefficient information 210 isled to a variable length encoder 111.

Further, the quantized orthogonal transformed coefficient information210 is input to a local decoder. In this local decoder, the quantizedorthogonal transformed coefficient information 210 is subjected to aprocess to be inversive to that of the quantizer 104 and orthogonaltransformer 103 with a dequantizer 105 and an inverse orthogonaltransformer 106, to reconstruct a signal similar to the predictive errorsignal. Thereafter, the reconstructed signal is added to a predictiveimage signal 212 input through a switch 109 with an adder 107 togenerate a local decoded image signal 211. The local decoded imagesignal 211 is input to a frame memory/predictive image generator 108.

The frame memory/predictive image generator 108 selects one combinationfrom a plurality of combinations of a reference frame number and apredictive parameter that are prepared beforehand. The image signal(local decoded image signal 211) of the reference frame indicated by thereference frame number of the selected combination is subjected tocomputation for a linear sum according to a predictive parameter in theselected combination. Further, in this example, a reference image signalof a frame unit is generated by adding an offset to the linear sumaccording to the predictive parameter. Thereafter, the framememory/predictive image generator 108 subjects the reference imagesignal to motion compensation using a motion vector to generate apredictive image signal 212.

In this process, the frame memory/predictive image generator 108generates motion vector information 214 and index information 215indicating selected combination of the reference frame number andpredictive parameter, and further sends information to be necessary forselection of an encoding mode to the mode selector 212. The motionvector information 214 and index information 215 are input to thevariable length encoder 111. The frame memory/predictive image generator108 will be described in detail later.

A mode selector 110 selects an encoding mode in units of a macroblockbased on the predictive information P from the frame memory/predictiveimage generator 108, that is, selects either of an intraframe encodingand a motion compensated predictive interframe encoding to output switchcontrol signals M and S.

In the intraframe encoding mode, switches 102 and 112 are switched to Aside by the switch control signals M and S, whereby the input videosignal 100 is input to the orthogonal transformer 103. In the interframeencoding mode, the switches 102 and 112 are switched to a B side by theswitch control signals M and S, so that the predictive error signal fromthe subtracter 102 is input to the orthogonal transformer 103, and thepredictive image signal 212 from the frame memory/predictive imagegenerator 108 is input to the adder 107. The mode information 213 isoutput from the mode selector 212 and input to the variable lengthencoder 111.

In the variable length encoder 111, the orthogonal transformedcoefficient information 210, mode information 213, motion vectorinformation 214 and index information 215 are subjected to a variablelength encoding. Each variable-length code generated in this way ismultiplied and then filtered by the output buffer 115. Thus, the encodeddata 116 output from the output buffer 115 is sent to the transmissionsystem or a storage system not shown.

The encoding controller 113 monitors control of the encoder 112,concretely, for example, a buffering volume of the output buffer 115,and controls an encoding parameter such as quantization step size of thequantizer 104 for the purpose of making the buffering volume constant.

(With Reference to the Frame Memory/Predictive Image Generator 108)

FIG. 2 shows detailed configuration of the frame memory/predictive imagegenerator 108 in FIG. 1. In FIG. 2, the local decoded image signal 211input from the adder 107 in FIG. 1 is stored in the frame memory set 202under control of the memory controller 201. The frame memory set 202comprises a plurality of (N) frame memories FM1 to FMN for temporallystoring the local decoded image signal 211 as a reference frame.

The predictive parameter controller 203 prepares a plurality ofcombinations of a reference frame number and a predictive parameter as atable beforehand. The predictive parametric controller 203 selects acombination of a reference frame number of the reference frame used forgenerating a predictive image signal 212 based on the input video signal100 and a predictive parameter, and outputs index information 215indicating the selected combination.

A plurality of frame motion evaluators 204 each generates a referenceimage signal according to the combination of the reference frame numberand index information, which is selected with the predictive parametercontroller 203. The plurality of frame motion evaluators 204 evaluatesan amount of motion and a predictive error from this reference imagesignal and the input image signal 100, and outputs motion vectorinformation 214 which makes a prediction error minimum. A plurality offrame motion compensators 205 generate the predictive image signal 212by subjecting the reference image signal selected with the plurality offrame motion evaluator 204 every block to motion compensation accordingto the motion vector.

(With Reference to Generation of a Predictive Image)

The following equations (1), (2) and (3) each show an example of apredictive equation using a reference image number and a predictiveparameter which are prepared with the predictive parameter controller203. An example as shown in here shows predictive equations applied to acase that a to-be-encoded image as referred to as so-called P picture issubjected to a motion compensated prediction using a single referenceimage (reference picture) to generate a predictive image signal, and acase that a to-be-encoded image as referred to as so-called B picture issubjected to a motion compensated prediction using only one of tworeference images to generate a predictive image signal.

Y=clip((D ₁(i)×R _(Y)(i)+2^(L) ^(Y) ⁻¹)>>L _(Y) +D ₂(i))  (1)

Cb=clip((E ₁(i)×(R _(Cb)(i)−128)+2^(L) ^(C) ⁻¹)>>L _(C) +E₂(i)+128)  (2)

Cr=clip((F(i)₁×(R _(Cr)(i)−128)+2^(L) ^(C) ⁻¹)>>L _(C) +F ₂(i)+128)  (3)

Y indicates a predictive image signal of a luminance signal, Cb and Crindicate predictive image signals of two color-difference signals, andRY(i), RCb(i) and RCr(i) indicate luminance signal and pixel values oftwo color-difference signals of the reference image signal of index i.D1(i) and D2(i) indicate a predictive coefficient of the luminancesignal of the index i, and an offset, respectively. E1(i) and E2(i)indicate a predictive coefficient of the color-difference signal Cb ofthe index i and offsets respectively. F1(i) and F2(i) indicate apredictive coefficient of the color-difference signal Cr of the index i,and an offset respectively. The index i indicates a value from zero to(the maximum number of reference images −1) and is encoded everyto-be-encoded block (for example, every macroblock) and transmitted bythe video decoding apparatus.

Prediction parameters D1(i), D2(i), E1(i), E2(i), F1(i), and F2(i) arevalues determined between the video encoding apparatus and the decodingapparatus beforehand, or predetermined encoding units such as a frame, afield or a slice, and shared with both apparatuses by being encodedalong with encoded data and transmitted from the video encodingapparatus to the decoding apparatus.

The equations (1), (2) and (3) are predictive equations making itpossible to avoid a division process by choosing denominator of thepredictive coefficient multiplied by the reference image signal like twoexponentiation, namely, 2, 4, 8, 16, . . . , and compute by anarithmetic shift. By this, it is possible to avoid increase of acomputation cost by the division process.

In other words, when >> in equations (1), (2) and (3) assumes a>>b, itis an operator subjecting an integer a to an arithmetic shift by b bitsto the right. Function clip( ) is a clipping function which is set to 0when the value within ( ) is smaller than 0, and to 255 when it islarger than 255, and the integer of 255 is returned from 0.

LY is a shift amount of a luminance signal, and LC is a shift amount ofa color-difference signal. These shift amounts LY and LC use valuesdetermined with a video encoding apparatus and a decoding apparatusbeforehand. Alternatively, they are shared with both apparatuses bybeing encoded along with a table and coded data in an encoding unit suchas a frame, a field or a slice, which is predetermined in a videoencoding apparatus, and transmitted to the video decoding apparatus.

In the present embodiment, a combination table of a reference imagenumber and a predictive parameter, which is shown in FIG. 3 is preparedin the predictive parameter controller 203 in FIG. 2. This table is usedwhen the number of reference image is 1. In FIG. 3, the index icorresponds to the predictive image which can be selected every block.In this example, there are four kinds of predictive images incorrespondence with 0-3 of the index i. The reference image number is,in other words, the number of a local decoded image used as a referenceimage. The table shown in FIG. 3 includes predictive parameters D1(i),D2(i), E1(i), E2(i), F1(i), and F2(i) that are assigned to a luminancesignal and two color-difference signals in correspondence with equations(1), (2) and (3).

Flag is a flag indicating whether the predictive equation using thepredictive parameters is applied to the reference image numberdesignated by the index i.

If Flag is “0”, the motion compensated prediction is performed using thelocal decoded image of the reference image number designated by theindex i without using the predictive parameter.

If Flag is “1”, the motion compensated prediction is done by generatinga predictive image according to the equations (1), (2) and (3), usingthe predictive parameter and the local decoded image of the referenceimage number designated by the index i. The Flag information uses valuesdetermined with a video encoding apparatus and a decoding apparatusbeforehand. Alternatively, it is shared with both apparatuses by beingencoded along with a table and coded data in an encoding unit such as aframe, a field or a slice, which is predetermined in a video encodingapparatus, and transmitted to the video decoding apparatus.

In these examples, when the index is i=0 with respect to the referenceimage number 105, a predictive image is generated using the predictiveparameter. In the case of i=0, the motion compensated prediction isperformed without using the predictive parameter. As thus described,there may be a plurality of predictive schemes with respect to the samereference image number.

The following equations (4), (5) and (6) show an example of predictiveequations for a reference image number and a predictive parameterprepared with the predictive parameter controller 203 when a predictiveimage signal is generated using two reference images.

P=clip((W ₀(i,j)×R(i)+W ₁(i,j)×R(j)+2^(L−1))>>L)  (4)

W ₀(i,j)=2^(L) −W ₁(i,j)  (5)

$\begin{matrix}{{W_{1}\left( {i,j} \right)} = \left\{ \begin{matrix}{{clip}\mspace{14mu} 2\left( {\frac{{U\left( {n,i} \right)} \times 2^{L}}{U\left( {j,i} \right)},} \right.} & {{U\left( {i,j} \right)} \neq 0} \\{2^{L - 1},} & {{U\left( {i,j} \right)} = 0}\end{matrix} \right.} & (6)\end{matrix}$

Since the relation of the equation (5) is established, the equation (4)can be transformed as follows:

P=clip((R(i)<<L+W(i,j)×(R(j)−R(i))+2^(L−1))>>L)  (7)

$\begin{matrix}{{W\left( {i,j} \right)} = \left\{ \begin{matrix}{{clip}\mspace{14mu} 2\left( {\frac{{U\left( {n,i} \right)} \times 2^{L}}{U\left( {j,i} \right)},} \right.} & {{U\left( {i,j} \right)} \neq 0} \\{2^{L - 1},} & {{U\left( {i,j} \right)} = 0}\end{matrix} \right.} & (8)\end{matrix}$

This shows an example of a predictive equation for the case ofperforming a bidirectional prediction in the case of so-called Bpicture. Two indexes i and j exist, and R(i) and R(j) indicate thereference images corresponding to the indexes i and j respectively.Consequently, it is assumed that two information items i and j are sentas the index information. W (j, j) indicates a predictive coefficient inthe indexes i and j. The function U used for computation of a predictivecoefficient is a function representing an image-to-image distance, andU(i, j) represents a distance between the reference image designated bythe index i and the reference image designated by the index j. nindicates a position of an image to be encoded currently.

In the present embodiment, it is assumed that the more past image hasposition information of a smaller value. Consequently, if the referenceimage designated by the index i is more future in terms of time than thereference image designated by the index j, U (i, j)>0. If the indexes iand j represent the same reference image in terms of time, U(i, j)=0. Ifthe reference image designated by the index i is more past in terms oftime than the reference image designated by the index j, U(i, j)<0. WhenU(i, j) is 0, the predictive coefficient W assumes 2L−1.

Concretely, a temporal position relation between a to-be-encoded imageto be encoded currently and two reference images is expressed as shownin FIGS. 4 to 7 using the indexes i and j. FIG. 4 shows an example thatthe to-be-encoded image n is interpolated between the reference imagedesignated by the index i and the reference image designated by theindex j.

Tn, Ti and Tj express positions of the to-be-encoded image, thereference image designated by the index i, and the reference imagedesignated by the index j, respectively. The value increases toward theright. Consequently, the relation of Ti<Tn<Tj is established. Thefunction U used for computation of the predictive coefficient W isobtained by U(n, i)=Tn−Ti, and U(j, i)=Tj−Ti, where U(n, i)>0, and U(j,i)>0.

FIG. 5 shows an example wherein the reference image designated by theindex i and the reference image designated by the index j both are in apast position in terms of time than the to-be-encoded image n. In otherwords, U(n, i)>0, and U(j, i)≦0.

FIG. 6 shows another example wherein the reference image designated bythe index i and the reference image designated by the index j both arein a past position in terms of time than the to-be-encoded image n. Inother words, U(n, i)>0, and U(j, i)≧0.

FIG. 7 shows an example wherein the reference image designated by theindex i and the reference image designated by the index j both are in afuture position in terms of time than the to-be-encoded image n. Inother words, U(n, i)<0, and U(j, i)≧0.

L is a shift amount in the equations (4) to (8). The shift amount uses avalue determined between the video encoding apparatus and the decodingapparatus beforehand, or is transmitted from the video encodingapparatus to the decoding apparatus with being encoded along withencoded data in a predetermined encoding unit such as a frame, a fieldor a slice, and shared with both apparatuses. Further, the function ofclip2 in the equations (6) and (8) is a function for returning aninteger with limiting the maximum value and the minimum value of theweighting factors obtained by computation using a value (referred to asa value simply) within ( ) of clip2( ), that is, an image-to-imagedistance. A plurality of configuration examples concerning this functionclip2 are shown hereinafter.

A first configuration of the function clip2 is a clipping function thatmakes −2^(M), when the value is smaller than −2^(M), and (2^(M)−1), whenit is larger than (2^(M)−1). The integer not less than −2^(M) and notmore than (2^(M)−1) is returned. With such configuration, if the pixelis 8 bits, 9 bits are necessary for expression of the value of (R (j)−R(i)), and (M+10) bits are necessary for expression of the predictivecoefficient W. Therefore, it is possible to compute a predictive imagevalue with an operation precision of (M+10) bits. M assumes anon-negative integer not less than L.

A second configuration of the function clip2 assumes a function having arule that it is set at 2^(L−1) when the value is smaller than −2^(M),and at 2^(L−1) when the value is larger than (2^(M)−1), and returning aninteger not less than −2^(M) and not more than (2^(M)−1). With thisconfiguration, when a distance relation between two reference images isexceptional, all images can be subjected to an average prediction.

A third configuration of the function clip2 is a clipping functionsetting at 1 when the value is smaller than 1, and 2^(M) when the valueis larger than 2^(M), and a function returning an integer not less than1 and not more than 2^(M). The difference with respect to the firstconfiguration of the function clip2 is that the value of the predictivecoefficient W does not become negative, resulting in that the positionalrelation of the reference image is limited more. Consequently, even iftwo identical reference images are combined, it is possible to change aprediction based on the predictive coefficient W and an averageprediction to each other by inverting the ways designated by the indexesi and j as the relations of FIGS. 5 and 6.

A fourth-configuration of the function clip2 is a clipping functionsetting at 0 when the value is smaller than 0, and 2^(L) when the valueis larger than 2^(L), and a function returning an integer not less than0 and not more than 2^(L). With such configuration, the value of thepredictive coefficient W becomes always a non-negative value not morethan 2^(L), so that an extrapolation prediction is prohibited.Alternatively, either of two reference images is used for prediction ina bidirectional prediction, too.

A fifth configuration of the function clip2 is a clipping functionsetting at 2^(L)−1 when the value is smaller than 1, and 2^(L)−1 whenthe value is larger than 2^(L), and a function returning an integer notless than 1 and not more than 2^(L)−1. With such configuration, thevalue of the predictive coefficient W becomes always a non-negativevalue not more than 2^(L)−1, so that an extrapolation prediction isprohibited. Alternatively, it is used for an average prediction of tworeference images.

When a distance between two reference images is unknown or undefined,for example, when either of the reference images or both thereof arereference images for the background and the storage, the predictivecoefficient W assumes to be set at the value of 2L−1. The predictivecoefficient W can be beforehand computed in an encoding unit such as aframe, a field, or a slice. Therefore, even when a predictive imagesignal is generated with two reference images, the computation per pixelcan be completed by multiplication of one time.

P=clip(R(i)+(W(i,j)×(R(j)−R(i))+2^(L−1))>>L)  (9)

Equation (9) is another example modifying the equation (4). In theequation (7), the operation for subjecting R(i) to an arithmetic shiftby L bits to the left beforehand was needed. However, in the equation(10), the arithmetic shift is omitted by bringing out it from aparenthesis. As a result, it is effective that an operation amount canbe decreased by the amount of the arithmetic shift. Instead, theorientation of rounding when the shifting is done varies upon a largeand small relation of the values of R(i) and (R)j. Therefore, the sameresult as the equation (4) is not obtained.

The following equations (10) to (20) may be used replacing with theequations (4) to (8). This is a method similar to a method of generatinga predictive image using a single reference image, that is, a method ofgenerating a final predictive image by generating a predictive image ofa single reference image of the index i and a predictive image of asingle reference image of the index j and averaging them. Since the sameprocess routine as that using a single reference image can be used untila halfway step of the process, the method has an advantage of making itpossible to reduce an amount of hardware and an amount of codes.

P _(Y)(i)=(W ₀(i)×R _(Y)(i)+2^(L) ^(Y) ⁻¹)>>L _(Y)  (10)

P _(Y)(j)=(W ₁(j)×R _(Y)(j)+2^(L) ^(Y) ⁻¹)>>L _(Y)  (11)

P _(Cb)(i)=(W ₀(i)×(R _(Cb)(i)−128)+2^(L) ^(C) ⁻¹)>>L _(C)+128  (12)

P _(Cb)(j)=(W ₁(j)×(R _(Cb)(j)−128)+2^(L) ^(C) ⁻¹)>>L _(C)+128  (13)

P _(Cr)(i)=(W ₀(i)×(R _(Cr)(i)−128)+2^(L) ^(C) ⁻¹)>>L _(C)+128  (14)

P _(Cr)(i)=(W ₁(j)×(R _(Cr)(j)−128)+2^(L) ^(C) ⁻¹)>>L _(C)+128  (15)

Y=clip(P _(Y)(i)+P _(Y)(j)+1)>>1  (16)

Cb=clip(P _(Cb)(i)+P _(Cb)(j)+1)>>1  (17)

Cr=clip(P _(Cr)(i)+P _(Cr)(j)+1)>>1  (18)

$\begin{matrix}\begin{matrix}{{W_{0}\left( {i,j} \right)} = \left\{ \begin{matrix}{2^{L + 1} - {{clip}\mspace{14mu} 2\left( {\frac{{U\left( {n,i} \right)} \times 2^{L + 1}}{U\left( {j,i} \right)},} \right.}} & {{U\left( {i,j} \right)} \neq 0} \\{2^{L},} & {{U\left( {i,j} \right)} = 0}\end{matrix} \right.} & \;\end{matrix} & (19) \\\begin{matrix}{{W_{1}\left( {i,j} \right)} = \left\{ \begin{matrix}{{clip}\mspace{14mu} 2\left( {\frac{{U\left( {n,i} \right)} \times 2^{L + 1}}{U\left( {j,i} \right)},} \right.} & {{U\left( {i,j} \right)} \neq 0} \\{2^{L},} & {{U\left( {i,j} \right)} = 0}\end{matrix} \right.} & \;\end{matrix} & (20)\end{matrix}$

(With Reference to a Procedure for a Selection of a Predictive Schemeand an Encoding Mode Determination)

An example of a concrete procedure of a selection of a prediction scheme(combination of a reference image number with a predictive parameter)and an encoding mode determination every macroblock in the presentembodiment will be described referring to FIG. 8.

The assumable maximum value is set to a variable min_D (step S101).LOOP1 (step S102) shows a repetition process for use in selection of apredictive scheme in interframe encoding. Variable i represents a valueof the index shown in FIG. 3. The evaluation value D of each index(combination of a reference frame number with a predictive parameter) iscomputed from the number of encoded bits concerning motion vectorinformation 214 (the number of encoded bits of the variable-length codeoutput from the variable length coding device 111 in correspondence withthe motion vector information 214) and a prediction error absolute valuesum, so that the optimum motion vector can be derived every predictivescheme. A motion vector which makes the evaluation value D a minimumvalue is selected (step S103). This evaluation value D is compared withmin_D (step S104). If the evaluation value D is smaller than min_D, theevaluation value D is assumed min_D, and the index i is substituted inmin_i (step S105).

The evaluation value D in the case of intraframe encoding is computed(step S106). This evaluation value D is compared with min_D (step S107).If, as a result of this comparison, min_D is smaller, the mode MODE isdetermined to be an interframe encoding, and min_i is substituted inindex information INDEX (step S108). If the evaluation value D issmaller, the mode MODE is determined to be an intraframe encoding (stepS109). The evaluation value D assumes an estimated value of the numberof encoded bits at the same quantization step size.

(With Reference to the Decoding Side)

The video decoding apparatus corresponding to the video encodingapparatus shown in FIG. 1 will be described. FIG. 9 shows configurationof the video decoding apparatus related to the present embodiment. Codeddata 300 sent out from the video encoding apparatus of the configurationshown in FIG. 1 and passed through a transmission system or a storagesystem is saved in an input buffer 301 once. The coded data 300 of theinput buffer 301 is divided based on syntax by a demultiplexer 302 foreach frame, and then input to the variable length decoder 303. Thevariable length decoder 303 decodes the variable-length code of eachsyntax of the coded data 300 to reproduce quantized orthogonaltransformed coefficients, mode information 413, motion vectorinformation 414 and index information 415.

The quantized orthogonal transformed coefficient of the reproducedinformation is dequantized with the dequantizer 304. The dequantizedcoefficient is subjected to an inverse-orthogonal transformation withthe inverse orthogonal transformer 305. When the mode information 413indicates the intraframe encoding mode, a playback image signal isoutput from the inverse orthogonal transformer 305, and output as anultimate playback image signal 310 via the adder 306. When the modeinformation 413 indicates the interframe encoding mode, a predictiveerror signal is output from the inverse orthogonal transformer 305, andfurther a mode selection switch 308 is turned on. The playback imagesignal 310 is output by adding the predictive error signal and thepredictive image signal 412 output from the frame memory/predictiveimage generator 308 with the adder 306. The playback image signal 310 isstored in the frame memory/predictive image generator 308 as a referenceimage signal.

The mode information 413, motion vector information 414 and indexinformation 415 are input to the frame memory/predictive image generator308. The mode information 413 is input to the mode selection switch 309,which is turned on in the case of the interframe encoding mode andturned off in the case of the intraframe encoding mode.

The frame memory/predictive image generator 308 prepares for a table aplurality of combinations of the reference image number and predictiveparameter that are prepared similarly to the frame memory/predictiveimage generator 108 of the encoding side as shown in FIG. 1, and selectsone combination designated by the index information 415 from the table.The image signal (playback image signal 310) of the reference imagedesignated by the reference image number of the selected combination issubjected to a linear sum according to the predictive parameter of theselected combination. Further, the offset according to the predictiveparameter is added to the reproduced image signal 310. As a result, areference image signal is generated. Thereafter, the predictive imagesignal 412 is generated by subjecting the generated reference imagesignal to motion compensation using a motion vector indicated by themotion vector information 414.

(With Reference to Frame Memory/Predictive Image Generator 308)

FIG. 10 shows detailed configuration of the frame memory/predictiveimage generator 308 in FIG. 9. In FIG. 10, the reproduced image signal310 output from the adder 306 in FIG. 9 is stored in the frame memoryset 402 with control by a memory controller 401. The frame memory set402 comprises a plurality of (N) frame memories FM1 to FMN to savetemporally the playback image signal 310 as a reference image.

The predictive parameter controller 403 prepares as a table combinationof a reference image number with a predictive parameter as shown in FIG.3 beforehand, and selects combination of the reference image number ofthe reference image used for generation of the predictive image signal412 with the predictive parameter based on the index information 415from the variable length decoders 303 in FIG. 9. A plurality of framemotion compensators 404 generates a reference image signal according tocombination of the reference image number with the index informationwhich is selected by the predictive parameter controller 403. Thepredictive image signal 412 is generated by subjecting the referenceimage signal to motion compensation in units of a block according to amotion vector indicated by the motion vector information 414 from thevariable length decoder 303 in FIG. 9.

(With Reference to Syntax of Index Information)

FIG. 11 shows an example of syntax when encoding index information ineach block. Each block has mode information MODE. It is determinedwhether index information IDi indicating a value of the index i andindex information IDj indicating a value of the index j are encodedaccording to the mode information MODE. After encoded index information,motion vector information MVi for motion compensated prediction of theindex i and motion vector information MVi for motion compensatedprediction of index i and index j is encoded as motion vectorinformation of each block.

(With Reference to the Data Structure of an Encoding Bit Stream)

FIG. 12 shows an example of a concrete encoding bit stream every blockwhen generating a predictive image using a single reference image. Indexinformation IDi is disposed following the mode information MODE andmotion vector information MVi is disposed thereafter. The motion vectorinformation MVi is usually two-dimensional vector information. Further,a plurality of two-dimensional vectors may be send depending upon themotion compensation method in the block that is designated by the modeinformation.

FIG. 13 shows an example of a concrete encoding bit stream every blockwhen generating a predictive image using two reference images. Indexinformation IDi and index information IDj are disposed following modeinformation MODE, and motion vector information MVi and motion vectorinformation MVj are disposed thereafter. The motion vector informationMVi and the motion vector information j are usually two-dimensionalvector information. Further, a plurality of two-dimensional vectors maybe send depending upon the motion compensation method in the block thatis designated by the mode information.

According to the present embodiment as discussed above, when apredictive image is generated using a single reference image, thepredictive image is generated by a linear prediction using a predictivecoefficient and an offset as the predictive parameters. This methodpermits generation of an appropriate predictive image for a field imagecorresponding to an image obtained by combining monochrome images. In amethod of selecting simply one combination from a plurality ofcombinations of a reference picture number and a predictive parameter,when there is a plurality of reference images, the multiplication perpixel must be performed a plurality of times. This increases an amountof computation. However, in the present embodiment, the necessarymultiplication may be one time per pixel.

On the other hand, when a predictive image is generated by means of tworeference images, the predictive image is generated by obtainingweighted mean of the two reference images, using a weighting factor andan offset that are obtained from a distance between the two referenceimages. This method makes it possible to generate an appropriatepredictive image for a dissolve image in which two images are mixed. Inthis time, if the equation used in the present embodiment is used, anecessary multiplication may be one time per pixel.

According to the present embodiment as thus described, an appropriatepredictive image can be generated for a feed image as well as a dissolveimage by one time of multiplication per pixel. Since the multiplicationcan be completed one time per pixel, a hardware scale and an operationcost can be decreased in both of an encoding side and a decoding side.

In the above description, the method of generating a predictive image ischanged according to the number of reference images. However, the methodof generating a predictive image may be changed in units of image or inunits of slice according to a difference of a prediction type asreferred to as so-called a picture type or a slice type. When onlyeither one of the reference images in the case of, for example, a Bpicture is used, the predictive image is not generated using thepredictive parameter, but the motion compensated prediction is carriedout using normal local decoded image.

A predictive image generating procedure using a method of changing apredictive image generating method according to a difference of thepredictive type as well as the number of reference images is describedwith reference to FIG. 14. In this example, a method of generating apredictive image is changed in units of slice.

A predictive type (referred to a slice type) of a to-be-encoded slicewhich is a to-be-encoded region is determined, and it is divided intothree slices, that is, an I slice in which a to-be-encoded slice issubjected to an intraframe encoding (intraframe prediction), a P slicewhich is predicted using a single reference image (one way prediction),and a B slice which is predicted using maximum two reference images(bidirectional prediction) (step S201).

If the to-be-encoded slice is an I slice as a result of determination ofstep S201, the intraframe encoding (intraframe encoding) is done (stepS202). If the to-be-encoded slice is a P slice, a predictive schemebased on combination of one reference image and a predictive parameteras described above is adopted (step S203).

If the to-be-encoded slice a B slice, the number of reference images ischecked (step S204), and the predictive scheme is changed according tothe result. In other words, if the to-be-encoded slice is a B slice, andthe reference image is single, normal motion compensative prediction isadopted (step S205). If the to-be-encoded slice is a B slice, and tworeference images are used, a predictive scheme corresponding to animage-to-image distance of the two reference images is adopted (stepS206).

Second Embodiment

There will be explained a second embodiment of the present invention.The entire configuration of the video encoding apparatus and the videodecoding apparatus in the present embodiment is approximately similar tothat of the first embodiment. Accordingly, only a difference withrespect to the first embodiment is described. The present embodimentshows an example when the first embodiment is combined with otherscheme.

The following equation (21) is a predictive equation of a bidirectionalprediction of so-called B picture using two reference images, and afirst method of averaging the motion compensative predictive image oftwo reference images simply.

P=(R(i)+R(j)+1)>>1  (21)

In the first method, change information (change flag) for changingbetween a predictive equation shown by either of equations (4) to (6),equations (7) to (8), equation (9) or equations (10) to (20) and apredictive equation shown by an equation (21) are encoded along with theencoded data in a predetermined encoding unit such as a picture, aframe, a field and a slice, and transmitted from a video encodingapparatus to a decoding apparatus to be shared with both apparatuses. Inother words, a predictive equation shown in either of the equations (4)to (6), equations (7) and (8), equation (10) or equations (10) to (20)and a predictive equation shown by the equation (21) are changed asneeded.

According to the first method, a weighted mean depending on animage-to-image distance and a simple average of the reference images canbe adoptively changed, whereby the improvement of a predictive efficientcan be expected. Because the equation (21) includes no multiplication,an computational amount is not increased.

The equations (22) to (27) and equations (28) to (33) show a method ofgenerating a predictive parameter for two reference images using apredictive parameter when the reference image is single. The presentembodiment shows an example which these methods are combined with thefirst embodiment. At first, the equations (22) to (27) show a secondmethod of obtaining a predictive value by averaging the value of thepredictive equation when the reference image is single.

P _(Y)(i)=(D ₁(i)×R _(Y)(i)+2^(L) ^(Y) ⁻¹)>>L _(Y) +D ₂(i)  (22)

P _(Cb)(i)=(E ₁(i)×(R _(Cb)(i)−128)+2^(L) ^(C) ⁻¹)>>L _(C) +E₂(i)+128  (23)

P _(Cr)(i)=(F(i)₁×(R _(Cr)(i)−128)+2^(L) ^(C) ⁻¹)>>L _(C) +F₂(i)+128  (24)

Y=clip((P _(Y)(i)+P _(Y)(j)+1)>>1)  (25)

Cb=clip((P _(Cb)(i)+P _(Cb)(j)+1)>>1)  (26)

Cr=clip((P _(Cr)(i)+P _(Cr)(j)+1)>>1)  (27)

PY(i), PCb(i) and PCr(i) are results in the middle of a predictive valueof luminance signal Y, color-difference signal Cb and color-differencesignal Cr respectively.

In the second method, change information (change flag) for changingbetween a predictive equation shown by either of equations (4) to (6),equations (7) to (8), equation (9) or equations (10) to (20) and apredictive equation shown by equations (22) to (27) are encoded alongwith the encoded data in a predetermined encoding unit such as apicture, a frame, a field and a slice, and transmitted from a videoencoding apparatus to a decoding apparatus to be shared with bothapparatuses. As thus described, a predictive equation shown in either ofthe equations (4) to (6), equations (7) and (8), equation (9) orequations (10) to (20) and a predictive equation shown by the equations(22) to (27) are changed as needed.

According to the second method, a weighted mean depending on animage-to-image distance and a predictive image based on a linearprediction using two reference images simple can be adoptively changed,whereby the improvement of a predictive efficient can be expected.However, according to the predictive type shown in the equations (22) to(27), although the multiplication number per pixel is twice, there isthe merit that the degrees of freedom of the predictive coefficientincreases. Therefore, further improvement of the predictive efficiencycan be expected.

Equations (28) to (33) show as another predictive equation an example ofa linear predictive equation using two reference images generated usingtwo predictive parameters in the case of a single reference image.

P _(Y)(i,j)=(D ₁(i)×R _(Y)(i)+D ₁(j)×R _(Y)(j)+2^(L) ^(Y) )>>(L_(Y)+1)+(D ₂(i)+D ₂(j)+1)>>1  (28)

P _(Cb)(i,j)=(E ₁(i)×(R _(Cb)(i)−128)+E ₁(j)×(R _(Cb)(j)−128)+2^(L) ^(C))>>(L _(C)+1)+(E ₂(i)+E ₂(j)+1)>>1+128  (29)

P _(Cr)(i,j)=(F ₁(i)×(R _(Cr)(i)−128)+F ₁(j)×(R _(Cr)(j)−128)+2^(L) ^(C))>>(L _(C)+1)+(F ₂(i)+F ₂(j)+1)>>1+128  (30)

Y=clip(P _(Y)(i,j))  (31)

Cb=clip(P _(Cb)(i,j))  (32)

Cr=clip(P _(Cr)(i,j))  (33)

In the third method, change information (change flag) for changingbetween a predictive equation shown by either of equations (4) to (6),equations (7) to (8), equation (9) or equations (10) to (20) and apredictive equation shown by equations (28) to (33) are encoded alongwith the encoded data in a predetermined encoding unit such as apicture, a frame, a field and a slice, and transmitted from a videoencoding apparatus to a decoding apparatus to be shared with bothapparatuses. As thus described, a predictive equation shown in either ofthe equations (4) to (6), equations (7) and (8), equation (9) orequations (10) to (20) and a predictive equation shown by the equations(28) to (33) are changed as needed.

According to the third method, a weighted mean depending on animage-to-image distance and a predictive image based on a linearprediction using two reference images can be adoptively changed, wherebythe improvement of a predictive efficient can be expected. However,according to the predictive type shown in the equations (28) to (33),although the multiplication number per pixel is twice, there is themerit that the degrees of freedom of the predictive coefficientincreases. Therefore, further improvement of the predictive efficiencycan be expected.

The above embodiment is explained as an example of a videoencoding/decoding system using orthogonal transformation in units ofblock. However, the method of the present invention described in theabove embodiment can be adopted to a case with the use of anothertransformation technique such as wavelet transformation.

A process of video encoding and decoding concerning the presentinvention may be realized as a hardware (apparatus), and may be executedby a computer using software. A part of the process may be realized witha hardware, and the other thereof may be executed by software.Consequently, according to the present invention, there can be provideda program to make a computer execute a video encoding or a decodingprocess or a storage medium stored the program.

As discussed above, according to the current invention, a videoencoding/decoding can perform with a high efficient and a littlecomputational amount by doing an appropriate prediction for a videoimage whose luminance varies in terms of time such as a fading image anda dissolve image in particular.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. A video decoding method for subjecting a to-be-decoded block of ato-be-decoded image to motion compensated prediction decoding,comprising: a first step of decoding encoded data of a to-be-decodedimage to obtain a quantized orthogonal transform coefficient, motionvector information and index information; a second step of calculating afirst distance between a first reference image designated by the indexinformation and the to-be-decoded image and a second distance betweenthe first reference image and a second reference image designated by theindex information; a third step of calculating a first weighting factorfor the first reference image and a second weighting factor for thesecond reference image based on a ratio of the first distance withrespect to the second distance; a fourth step of generating a predictionimage by calculating a linear sum of an image of a region specified bythe motion vector information in the first reference image and an imageof a region specified by the motion vector information according to thefirst weighting factor and the second weighting factor; a fifth step ofobtaining a prediction error by subjecting the quantized orthogonaltransform coefficient information to inverse quantization and inverseorthogonal transform; and a sixth step of generating a reproductionimage by adding the prediction error and the prediction image; whereinthe third step sets values of both of the first weighting factor and thesecond weighting factor to a value for an average prediction if (1) thesecond weighting factor is larger than an upper limit, or (2) the secondweighting factor is smaller than a lower limit, or (3) the seconddistance is 0, or (4) at least one of the first image and the secondimage is a long term reference image.
 2. A video decoding apparatus ofsubjecting a to-be-decoded block of a to-be-decoded image to motioncompensated prediction decoding, comprising: a first unit configured todecode encoded data of a to-be-decoded image to obtain a quantizedorthogonal transform coefficient, motion vector information and indexinformation; a second unit configured to calculate a first distancebetween a first reference image designated by the index information andthe to-be-decoded image and a second distance between the firstreference image and a second reference image designated by the indexinformation; a third unit configured to calculate a first weightingfactor for the first reference image and a second weighting factor forthe second reference image based on a ratio of the first distance withrespect to the second distance; a fourth unit configured to generate aprediction image by calculating a linear sum of an image of a regionspecified by the motion vector information in the first reference imageand an image of a region specified by the motion vector informationaccording to the first weighting factor and the second weighting factor;a fifth unit configured to obtain a prediction error by subjecting thequantized orthogonal transform coefficient information to inversequantization and inverse orthogonal transform; and a sixth unitconfigured to generate a reproduction image by adding the predictionerror and the prediction image; wherein the third unit sets values ofboth of the first weighting factor and the second weighting factor to avalue for an average prediction if (1) the second weighting factor islarger than an upper limit, or (2) the second weighting factor issmaller than a lower limit, or (3) the second distance is 0, or (4) atleast one of the first image and the second image is a long termreference image.